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ABSTPACT 

Described are the formative evaluation procedures 
used to assess materials to teach educable mentally retarded children 
concepts and* skiLis in the areas of money, measurement, and time. It 
is explained th-afc "formative" evaluation refers to assessment of an 
instructional product during its development, in order , to identify 
wa-ys in which the materials can be modified. Four stages of^ the 
overall project evaluation plan- are outlined. Aspects of the * 
formative evaluation design focused on include the six instructional 
variables assessed, such as instructional effectiveness, the sources 
of input, such as a -^eachei? review board. Appended is a sample ^ 
evaluation form for teacher use. (LS) 
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The University of Minnesota Research, Development and Demonstration 
Center in Education of Handicapped Children , has been . established to 
concentrate on intervention strategies and materials which develop and 
iinprol?e language and communication skills in young handicapped children. 

The long term objective of the Center is to improve the language 
and communication abilities of handicapped children by means of iden- ^ 
^ - tification of linguistically and poteoti^hUy linguistically haadicapped 
children, development and evaluation of intervention strategies with 
young handicapped children and dissemination of findings and products 
of benefit to young handicapped children. * 
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The Forma-tive Eviluatlon Design of ^ the Vocabulary Development Project 

Patricia H. Kr^, Martha L. Thurlow, James E. Turnure, 
/ Arthur M. Taylor, Roseshel Howe 

>. X • " University of Minnesota 

< J ; 

The evaluation of materials used In the classroom Is an Impor- 
tant educational responsibility which is frequently ignored, CronbacK 
(1963) has defined educational evaluation as "the collection and use 
of information to make decisions about an educational program," This 
general definition has been the basis for several evaluation endeavors 
over the past decade. Generally, however, these evaluation attempts 
have been concerned only with verifying the effectiveness of the final 
version of an educational product. Evaluation attempts during the 
devellbpment stage have been relatively rare. 

Scrlven (1967) has made a distinction between two types of eval- 
uation mdd^s - "formative" evaluation and "suramatlve" evaluation. 
^*Formatlve"\evaluatlon jefers to the ass.essment of an instructional 
product durlny its development|. Its goal is to identify^ the ways in' 
which the materials [can be modified in order to optimize their effect- 

*iveness before tijie summatlve evaluation is undertaken. "Summatlve" 

\ 

evaluation ref ers\ to the assessment procedures occurring after the 



development of the\lnstructloffal product 



product has been complete. According 
to Scrlven, summatlve evaluation is based upor^ the' final product's 
use in a field-test situation, where its worth is compared to other 
products which attempt 'to accomplish similar goals. 
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Both. the formative and suminative aspects of evaluation are 
necessarily -important in the assessment of . edur^tional materials. 



Tne need for product evaluation to protect school children, the 

"largest single group of unprotected consumer^," has been noted by 

Cass (1973). Too often such children are required to learn from . 

^materials tnat have not been f ield^tested. Educational Products 

Information Exchange Institute (1972) has estimated that 99% 

of the materials used in schools have not been field-tested in 

— - a way tnat involves tne actual use. of the laaterrals in the ' 

s 

- classroom, assessment, and subsequent systematic revision 

where needed (Cass, 1973). The need for more of this type of for- 
^ mative evaluation has been emphatically proposed by Sandets and 
• Cunningham (1973). who argue tnat it is the most urgent need in the" 
^arsa of product development, 

Recently, materials were developed by the VoiabuWry Develop- ' 
ment Project- to teach educationally handicapped childrerK important 
concepts and skills in the areas of money, measurement and- time. 
In evaluating the materials, the Project observed Scriven's (1967) 
' distinction between formative and summative evaluation, and employed 
an expanded four-stage overall evaluation plan which included both 
types of evaluation. The overall evaluation plan, included a chain 

> 

of activities, ranging from >^sio research to" the application of , 
this research in 'the development process; the pilot-testing of 
initial materials, with feedback from a wide-range of sources; re- 
Vision of the materials; and finally, a large-scale field-test ^of ' 
. the fevised materials. 

\ 
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Perhaps the most intensi^"^ aspect of this overall evaluation- 
plan involved the formative evaluation of the materials as they 
were developed. The goal of the formative evaluation of each in- 
structional unit was -to modify the materiaj^,^ order to optimize 
their -effectiveness for educable mentally retarded (EMR) children, 
th^ population for whom the Program was developed. The success of 
the revised materials -^as to be assessed- in the extensive summa- 
tive evaluation of the materials, which was to occur in urban, 
rural, and suburban communities in the 'state of Minnesota. 

Each df the five units in the Mpney, Measurement and Time 
Program underwent extensive firmative evaluation so that revisions 
could be made and tested in a\^at±ve evaluation design. The 
purpose of this paper is to explain how the formative evaluation 
design was incorporated ipto the overall evaluation plan of the 
Vocabulary Development Project, and specifically, to describe the 
' formative evaluation aesign in detail so that others 'might develop 
a similar evaluation plan to meet their own needs. 

Overall Evaluation Plan of the Vocabulary Development Project 

The goal of the overall evaluation plan of the Vocabulary 
Development Project was to identify effective instructional tech- " 
niques and to incorporate these into materials which would meet ^ 
some of the instructional n^eds of young EMR children. An overall 
.evaluation plan which incorporaCed each of these goals was developed. 
It was the belief of the Project that evaluation would not only 



facilitate the process of Instructional development, but that It 
'would also prove to be an Integrail part of ^that process: 

Four stages were Identified In the Project's overall evaluation, 
plan (see Figure 1). Stage One Involved the movement from "research ' 
to development." In this stage', relevant findings from basic 
research were Identified, analyzed, and summarized, and then trans- 
jlated Into prototype classroom materials. Much of the research foun- 
dation came from elaboration resejjrch conducted by the Project's 
directors (cf., Taylor, Josbergeir, & Knowlton, 1972; Thurlow & Tur'nure, 
1972; Turnure, 1971; Tumure & Thurlow, 1973a, b; Turnure & Walsh, 
19-71; Whlteiy & Taylor, 1973) and from other research being conducted 
at the University of Minnesota's Research, Development and Demons tra- 
tlon Center . , Bender & Taylor, 1973; Danner & Taylor 1973 ; Riegel 
Dann^r & Taylor, 1972). Relevant findings from numerous other Indi- 
vldualfTworklng In the area of elaboration were also reviewed and 
Incorporated during this stage (cf . , Ammon & Ammon , 1971; Bower, 
1970; Rohwer, 1970). It was In this Initial stage that prototype 
materials were written by taking the findings from basic learning 
stra'tegies research and translating them into materials to be used 
in the classroom. The role of the evaluator during this preliminary 
stage to clarify and refine the objectives of the materials, 
making explicit the underlying rationale. 

Stage Two Involved the assessment of the prototype materials in 
the classroom. Several "experimental" studies contributed to this 
stage of the Project's overall evaluation design (cf.. Bender, 



/ 

i 



Figure 1 

Overall Evaluation Plan of the Vocabulary, Development Project 



Stage I: Research, to Development 

Relevant basic research findings were identified, 
analyzed , and summarized . These were translated 
into prototype classroom materials. 



Stage 11:, Assessment of Prototype Materials 

Prototype materials were evaluated in experimental 
studies. Effective instructional approaches werd 
selected for the future development of instructional 
units, and the instructional needs of the target 
population were identified. ^ 



Stage III: Formative Evaluation 

Instructional units were develc^ed and assessed 
in accordance with a formative evaluation design. 
Feedback from several sources on specific aspects 
-of the Instruction allowed for revision of the 
materials. 



\ 




Stage IV: SummaftL^^Evaluation 



The revised materials were field-tested to deter- 
mine their effectiveness in comparison to other 
instructional products. Recommendation for revis- 
ions and standardization of testing instruments 
-concluded this stage. 



fK 



Taylor, Rlegel, & Turnure, 1972; Rlegel & Taylor, 1973: Rlegel, 
Ta/^lor, Clarren, & Danner, 1972). In particular, one "experimental" 
study compared alternative instructional approaches for teaching 
vocabulary, and evaluated the children's instructional needs in light 
of the limitations of each approach. The details of this study, 
which perhaps contributed most directly to the development of the 
Money, Measurement and Time Program, are described fully in Taylor, 
Thurlow, and Tur nure (1974) , along with the rationale for the decis- 
J.on to develop a program of instruction on money, measurement and 
time concepts. 

Stage Three involved the development of the first version of 
the Money, Measurement and Time Program (initially referred^ to as 
the "Math Vocabulary Project;" cf., Taylor, Thurlow, & Turnure, 
^ 1973), the pilot-testing of the materials, and their revision. 
This stage represented the formative evaluation of the materials. 
As each unit was developed, it was tested in several classrooms 
(from four to six) of EMR children. During this stage, there was 
maximal interaction between the developers and the class tooms using 

the, materials . Teachers assessed the materials daily on teacher 

/ ' 

evaluation forms', observers visited the qlassrooms, and the chil- 

^ J 

dren were tested frequently throughout the instruction. The care- 

. . J ! 

ful m^)nitoring of tHe materials, with the immediate feedback from 

teachers, obs^ervers, and test results enabled the materials developers 

to quickly and efficiently revise and modify ^ach unit. In the 

development of the Money, Measurement and Time Program, tliis stage 



of formative evaluation was perhaps the most important since the 



developers were gathering data about the product's effectiveness 
for the specific purpose of making major revisions where weaknesses 
were observed* 

Stage Four involved the field-testing of the revised materials 
on a relatively large-scale, with minimal interaction between the 
developers and the classrooms. During the Stage Four field-test, 
the performances of groups receiving the Money, Measurement and Time 
Program were compared to those of "Hawthorne" control groups. In 
addition, the effectiveness of the Money, Measurement and Time Pro- 
gram was compared with that of materials typically used by the 

teachers (often these included materials developed by the teachers 
« 

themselves) in their EMR classrooms. The goal of the evaluator 
during this stage yas ^to standai;^ize the testing instruments, to 
examine the attainment ,of objectives, and to document pretest, to 
posttest gains. This stage, referred to as the summative evaluation 
by the Vocabulary Development Project, differed in two* respects from 
the **summative evaluation" described* by Scriven (1967), First, 
the product was not assessed in comparison to a specific alternative 
product that attempted to accomplish similar goals. The decision 



to delete this aspect was made by Project Directors after failing 



range of content as the Money, Measurement and Time Program, ^Se6ond, 
because the Money, Measurement and Time Program had not yet gone to 



a publisher, it was still possible to make recommendations for chang 




to find an appropriate alternative product which covered the sam6 
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"-before publication of the final product. 

' The present paper Is a discussion of Stage Three of the over- 
all evaluation plan. This stage Involved the formative evaluation 
of each of the Instructional units which had been produ(?led on the 
basis of the first two stages In the overall plan. Five las true- 
tlonal units underwent formative evaluation: Money, Measurement 
of Length, Measurement of Weight, Time with the Clock, and Time 
with the' Calendar (of., Thurlow, Taylor, & Tumure, 1973). The 
basic design of the f ormatlve^valuatlon for ea:ch of the units was 
the same. 

Formative Evaluation Design of the ^Vocabulary Development Pfo^ject 

As Indicated previously, the goal of the formative evaluation 
design was to evaluate the materials as they were being developed 
so they could be^modlfled In a way that would optimize their 
effectiveness and useablllty. The Ideal was to obtain "rich," but 
cost-effective Information about the materials. It was hoped 
that the results of the evaluation yould be "expository,*' as des- 
cribed by Stake (1967): 

A full evaluation results in a story, supported 
perhaps by statistics and profiles. It tells 
what happened. It reveals perceptions and judg- 
ments that different groups and individuals 
hold — obtained, I^hope, by objective mean§. 
It tells of merit and shortcoming [p. 5] * 

Thus, opinions of various 'individuals, as well as ft'ard data, were 

consi^dered to Ije a crucial part of the evaluation design. 

. The formative evaluation design Involved a matrix of input " 

1-t 
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sources and instructional variables '(see Table 1). Each source 
> of input (e.g., classroOim observation') was used to assess one-or 
more variables (e.g., effectiveness, content, design). 

For the Money ^svl%asurement and Time Program, six instructional 



variables were Identl^ed as most germane to the success of the 

units in the Program, Thifese we rjfe : 1) Instructional effectiveness, 

2) Sequencing, 3) Content, 4) Design of the materials, 5) Need, fo-r 
■ t 

the instruction, and 6) Adequacy of testing procedures. 

Instructional/effectiveness involved the assessment of the 
subjects*' masbery of the objectives specified for the instruction. 
Where instruction was fouAd to be ineffective or/'weak, close scrutiny 
of the associated instructional techniques, seauencing, and the 
design of the materials was undertaken, in adydition to the inves- 
tigation of the specific content itself. 'The goal of 80% mastery 
by posttesting was set up as the criterion during pilot-tjesting . 
Instructional effectiveness was primarily determjjn'ed by^lllre'"analysis 
of data from tests given during instruction and f ram posttests . A 
secondary rating of instructional ef fectivenes^M^as obtained from 
teacher feedback an^classroom observation feedback. 

Sequencing o^' the ins truaticJiT referred to the flow from gne 

lesson to the next-r'ffom orre concept or 9kill to the- next, and 

/ 

from„©nC^tep in the development of a particular concept or skill 
to th^ next step. Sequencing in the initial version of each unit 
was determined primarily by a process Stake (1970) "has labeled 
"logical analysis." In other words, sequencing was based on the 



/ 
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Table 1 



The Formative Evaluation Design: A Matrix of 
Input Sources and Instructional Variables 



Instructional Vari ables 

: ~ 



Sources of Input 

Project personnel 

1. Project directors 

2. • Developers and technlcgi writers 

3. Outside Consultants 



Teacher Review Board ' 

1. Unit summdry meeting 

2. -Wr^lttfe evaluations 



Classroom Observations 



Data Analysis 



1. Diagnostic pretest 
\ 

2.1 Progress^test during lnstruct:lon 
3. Posttest results 



Other research & Instructional materials 
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** Indicates that the source was a very Important Input^^dssessl^g the 
Instructional process. 

* Indicates/ that the source was an Important input for assessing the 
Instruc^onal process. 
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^ feelings and knowledge of the Project .directors and the materials 
developers^ The formative evaluation process thus served to verify 
the sequence or provide a rationale for revising It. 

A major quantitative method by which sequencing was ascertained 
during the formative evaluation was order analysis (Krus, 1973, 
1974; Krus & Baft/ 1974; Taylor, Bart, & Howe, 1974). Order analy- 
sis of the posttest items, for instance, ^equenced them according to 
the order in which .students obtained mastery of them. Tree-like 
structures of the items, leading to the most difficult item showed ^ 
the paths and order in which instruction on the objectives should 
be presented for optimal learning. 

Content referred to the "substance" of the instruction. It 
Included, for example, the behavioral objectives, the specific words 
and skills, the pictures and wordings, and the tape-recorded voices 
which were used during the instruction. 

The behavioral objectives were, of course, a major factor con- ' 
sidered under content. Achievement related to all specified objec- 
tives, plus some incidental ones, was tested. In addition, personal 
judgments by teachers, consultants, and project pe^qonnel regarding- 
all objectives were also considered. Using as many sources of input ^ 
as possible, the evaluation design was set up to determine whether 
the objectives (and the higher-order purpose) for each lesson were: 
1) appropriate for the children, 2) consistent with, the instruction, 
3) the most important for the lesson, and 4) clearly stated. 

In the present evaluation design, "content" also referred to 
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the words and 'skills presented , ^and the Instructional techniques 

used to present them. The .Initial selection of words and skills 

was made on the basi^ of needs assessment, and presen^tlon within 

the instruction reflected the notion of the "growth of meaning.'* 

In other words, each vocabulary word was expand^ to encompass 

more and more meaning, and eventually rejated |skills were intror 

duced. For example, in the Mon^y Unit, each ^of^^^J;hV4J • S • coins 

was presented three times to develop three expansions of meaning 

(recognition, relative value, exact value). The formative evalu- 

ation procedures were designed specifically to assess this growth 

of meaning in the content, and especially the inclusion of -concepts 

needed before children could adequately proceed in the instruction 

(e.g., the need for prerequisite concepts like "more" and "most" 

before the presentation of relative value). 

Every aspect of content was assessed during the formative 
evaluation, and resulted in, perhaps, the most extensive amount of 
information collected concerning the materials being developed. 

Design of the materials related to th^general format used 
to present the instruction, Useability by both the children .and 
the teachers was considered. For example, the use of tape re- 
corded lessons, variations in the types of tape lessons, the 
frequency or location of "stop tapes" and^."pauses" during tape 
presentations^, and the size of the children's texts were just some 
of the aspects investigated in terms of useability and effectivenes 
for the children. The format of teacher's editionb^,. the use of 



step-by-step teacher-directed ^activities , and the inclusion of 
optional post-activities were a 'few of the factors considei;ed in 
teacher useability. The design of the materials was evaluated pri- 
marily by the Input from those who used the materials and those who 
developed them. ' 

Need for the Program was assessed prior to the initial develop 
ment of the Money, Measurement and Time Program*, and also during 
f ormative^^evaluation. A search of published instructional material 
was conducted to determine the need for the Program, This search 
failed to identify any relevant programs (i.e,, ones that provided 
instruction in the target content areas) that did not require entry 
skills such as reading and compujzation. Relevant materials were 
found to make assumptions about the child's entry level abjLlities, 
abilities which^ exceeded those of most educationally handicapped 
children. During formative evaluation, pretest data, as well as 
the input f^om teachers and a qualified math consultant, made 
it possible to more effectively document the need for instruction/ 

Field- testing procedures wer.e established on the.basisN of 
feedb^^^KTk from the formative evaluation. Although the repeated 
testing of objectives was an important aspect of the formative eval 
uation, this procedure was judged unnecessary and expensive for 
final field-testing of the materials. Therefore, the data from 
the formative evaluation (including data analysis and teacher feed- 
back) were used to construct an efficient and c'^ost-ef fective test 
for each unit. Other improvements in field-test procedures which 
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were the goal of the formative evaluation were: 1) the summar Izatlor 
of the test results In a concise and useable form for the teachers, 
2) the development of a short diagnostic placement test that could be 
administered by teachers, and 3) the analysis of transfer ^test^f 
highest potential payoff for the forthcoming summative .evaluation.^ 

The six Instructional variables discussed above were specific 
to the materials In the Money, Measurement and Time Program, although 
other materials certainly have 'similar Instructional variables which . 
could be evaluated. In order to adequately evaluate the Instructional 



variables during the development of materials, five basic sources of 
input? were identified and establlsfied by the Vocabulary Development-. 
Project (see Table 1), These input sources included: 1) Project* per- 
sonnel, 2) Teacher Review Boards, 3) Classroom observations, 4) Data 
analyses, and 5) Other research and instructional materials. 

Project; personnel included project directors, writers of the 
instruction, V^d outside consultants. The project directors we;:e* 
individuals who had been involved in both Stage One and Stage Two 

of the overall evaluation plan. The feedback from these individuals, 

^ 4 . 

based on their composite view of the data from other input sources,' 
was viewed as a major source of input to each of the six instruc- ^ 
tional variables, except the analysis of instructional effectiveness. 

The writers of t.he instruction were responsible for developing 
and revising the actual instructional materials . They were individuals 
who had been teachers of the target population, young EMR children-,, 
before being hired for the project. These individuals had been 
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Identified as particularly effective with these children ^nd had 
demonstrated an ablli^ty to write satisfactory materials based on 
.the general gilldellnes. and Jtjechnlcal specifications set up by project 
directors. The f eedbaclf, ^rom the writers was viewed as especially 
pertinent to the implementation and evaluatloa of sequencing, content 
and design of materials. ' ^* - ' ^ 

Outside consultants were* employed whenever possible to provide 
a^n "unbiased" source of feedback. Such individuals had academlp - 
and practical expertise with the content ^reas or with the target * 
subject population *(EMR children). To avoid *f he problems . of ten "en- 
countered with '^^ert" opinions ^(cf . , Armstrong 1973; Dershlmer, 
1968), the Vocabulary Development Project provlded^he outside cofT- 
sultants with a list. of the instructional variables df interest, 
along with a statement of the Program's purpose (cf., Taylor, Thurlow, 
& Turnure, 1973). The feedback from the outside consultants was 
u§^d to evaluate the need for the program, as wfell as lta;\ sequencing , 
content, jcieslgn, and evaluation proce"^ures. 

The Teacher Review Board was an Important feedback element ^et 
up to ofctaln maximal input from those individuals using the ufaterials. ^ 
The importance of including the teachers in the process of evaluation, 
especially formative evaluation has been noted by McLaughlin (1973). 
Not only is the teacher the main "consumer" of the materials beings 
evaluated, but the teacher has access to certain forms of data nether 
obtainable by^ testing lior visible to the classroom observer. Although 
the need for teacher evaluation is obvious, the best Instrupfents for 
obtaining such ev*aluafclons are not. Two methods were used^to obtain 
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feedback from teachers using the Money, Measurement and Time mater- 
. lals: 1) Written evaluations, and 2) Unit summary meetings. 

A separate Teacher Review board was set up .for eadh unit that 
underwent formative evaluation. Teachers were requested to participate 
and were informed Of their respohsibilities .(e.g. , to" teach the unit 

4 

daily, to provide- weekly lesson plans, to fill out evaluation forms, 
to allow for testing and classroom obseirvations , and finally, to 
attend a meeting following completion of the unit) . Teachers were 
modestly paid for completing the te.acher ^valuation forms and for 
attending the final Teacher Review Board meeting. Each teacher par- 
ticipating in the pilot-test was given an evaluation form for each 
lesson in the unit being tested. These very detailed ques'tionnaires 
(see Appendix 1) covered nearly every aspect o£ ' the instruction (e.g., 
cootent, preparation time ,. pictures , etc.),' and were generally about 
ten pag4s in length ./ ThUs , a great deal of data was amassed from 
the teacher's written evaluations. ^ , 

Fortunately, one~of the problems often encountered with the use 
of teacher evaluatfion forms (the noncompletion or nonreturn of forms^ 
cf., Latham, 1973) was rarely encountered under the present formative 
design, despite the relatively .lenghly evaluation Jorms used. This 
was. du|, in part, to the practice of paying teachers for compleg^ing 
the .forms. (Teachers were not paid'tor using the materials.) More 
importai^it, however, was the fact that the teachers were shown the' 
ways in which their written reactions were important and how they ^ 
formed a major source of input for the revision of the materials (cf . ' 



f. 



iri 



17 

arrangements, time of day, number of cKfldren, etc.). 

The classroom observations were also used, of course, to pro- 
vide additional feedback on the effectiveness of the Instruction. 
In all cashes, attempts were made to conduct observations when weak- 
nesses or problems were noted by the teachers, or when Project per- 
sonnel desired feedback on a specific lesson or activity, • 

Data analyses were a major source of input in the formative . 

evaluation design, especially for identifying areas where other 

f V » 1 ' 

aspects of instruction (e.g., content, sequencing, etc.) should under- 

go further scrutiny.. Data collected during formative evaluation 

included pretest data, pbsttegt data, and data from tests given while 

insrruction was in progpes's. 

Pretest data- were used to determine initial placement in a unit 

about to undergo formative evaluation r In all cases, teachers were 

♦ 

inforiped of the pretest performances of all children in their classes 
so that they would know where strengths and' weaknesses were. The main 
purpose of the pret^t data, however, was to provide a baseline for 
assessing performance levels after ins'truction. 

Diirlng instruction, ''post-lesson** tests were given frequently 
(after every two to' three lessons) to assess mastery on various objec- 
fives as relevant instruction was presented. These tests were used 
mainly to determine the Immediate "worth" of the lessons (l.e^, 
whether or not the objectives of a given lessonswere met Immediately 
after the instruction from that lesson waS presented). If 80%" mastery 
was not demonstrated on a given objective on a "post-lesson** test, 
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it was then included in the next "post- lesson" test to ascertain if 
later instruction affected performance. In addition, these tests ^ 
frequently included questions related to' objectives not yet taught • 
This was done in qrdeir to determine any*^ "transfer" that instruction 
might have to performance on future objectives.. Again, teachers were 
informed of "post-lesson" performance's so that they could modify in- * 
struction to cater to the strengths and weaknesses revealed. 

Post testing was conducted at the completion' of all instruction. 
All items were presen^d during the posttest to identify the "final" ' 
effects of the ins truction (including any drop-offs in pe^rformance 
after initial mastery). 

Since the instruction undergoing formative evaluation was designed 
to avoid making assumptions about the children's ability levels, each * 
objective in the instruction was tested two or three different ways 
to insure accurate measurement (Bart, personal communication) » This 
concurs with Dershimer's (1968) argument that evaluators should use 
more than a single measure in order to obtain richer data. Thus, in ' 

the* formative evaluatipn descri|bed here^ specific purposes of the 

i i 

instruction (e.g., to "know" aliout the U.Sw qoins) were tested in 



several different ways, (e.g.. Identify a coin when with others; name 
coin when iTy itself; describe c^in not in view; utilize name of coin 
in a sentence, etc.). In some cases, the tests included questions 
related to items not specified in the instruction. These were generally 
ones the teachers or project directors had identified as important or 
ones which the instruction might influence although not specifically 
being taught. . 
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Data collected during the formative evaluation of a particular 
unit were sununarlzed by project directors and assembled into a docu- 
ment which made specific rec6mmendations about all aspects of tlie 
instruction, except the design of the materials (see Table 1). These • 
recommendations not only described where instruction was weak, ,but also 
identified possible reasons and other sources to investigate before 
making any revisions. 

Other research and instructional materials were continually 

i 

reviewed during the formative evaluation of the Money, Measurement 
and Time Program. These outside sources were valuable in the con- 
tinuing evaluation of the need for t>he instruction and in providing, 
ideas for revising' the format ajid design of ^ the materials. 



Adoption of the Formative Evaluation Design 

\ Du^^g the Implementatibn of tl^e fomlative evaluation design, y 
oj the Vo'fcabulary. Development Project in« real-life classroom^ settir/g^^ 
the specific sources of in^ut and procedures for .obtaining feedback 
were sometimes modif^^d^ to 'conform to budget and time *requirementsf^. 

Amazingly, however, the "ideal" ;4esign was followed relatively closely 

^ . ^ J - / 

in the formative evaluation of all five units in the Money, Me'as^re- 

inent and Time Program (cf., Krus, Thurlow, Howe,Vaylor, \ Tunlure, 

1974; Thurlow, Krus, Howe, Taylor, & Turfiure, 1974 a,b,c). The prcv 

posed matrix of iT\put sources and instructional variables proved to 

be most effective and efficient for conducting formative evaluations. 

Perhaps the one aspect of this formative evaluation design^ 

which made it so successful was the extensive interactions that took 
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place between the various input sources, and the valuable composite 
picture of the materials. For instance, during the Teacher Review 
Board meetings, the pilot-test teachers freely discussed their reac- 
tions to the materials, even when project personnel indicated that 
they did not agree with the" teachers. The completeness of the test 
data, and the fact that feedback about the data was always given to 
^the teachers, allowed the teachers to search for reasons, either in 
their teaching methods or in the instruction ijtself , for the good or 
poor performances of the children in their 'classes. Projec-t personnel 
frequently made classroom observations", and were always aware of the 
current states of "the materials in the 'classroom. Although the matrix 

depiction of the formative evaluation design does not reflect the 

? 

varied and numerous interactions which occurred, they ^^re 'extremely 
important to the success of the* evaluation plan. 

Although the formative evaluation design presented here was 
developed'specificaliy for the Vocabulary Development Project to eval- 
uate and revise the vocabulary matferials it* was developing, the 
design seems to be one which could be easily adopted by any project 
peeking to engage in formative evaluation. 
.. The specification of the instructional variables of ii^terest, of 

•course, would vary with the product being evaluated. General factors, 
such as sequencing, content, design of the materials, instructional 
effedtlveness, testing instruments, and needs assessment seem to be 
non-specific enough to apply to almost any instructional product. 
The definition of these variables and the further delineation of them 

-(e.g., what does "content" encompass), however, would depend upon 
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t;|ie specific instructional product being evaluated. For example, 

y/^i^ce the Money, Measurement and Time Program was concerned with 

'< t 

vocabulary development, investigation of word content was crucial 
to* the revision. Needless to say, relevant instructional variables 
shbitld be specified regardless of the specific evaluation plan being 
'^^i,E(>|)lemented. 

Vj^"' ^® sources of input proposed in the present formative evalua- 

,»'*>' 

^|t^On design seem to be especially effective, and applicable to 

I^J-D'ost any project engaged in the development and evaluation of an 

' ' ' ^ * ' 

instructJ.onal product. Classroom observations, opinions, from pilot- 
tes^: teachers and project personnel, data analysis, and the con- 
tinued asse^merit of related research and instiructional materials 
are sources available to any developmental endeavor. Ae extensive 
use of these sourceSKand the attempt to provide opportunities for 
interactions between them appeared to be the key to the success of 
the formative evaluation design described here. 

^ysteniatic and frequent classroom observations provide an 
"outside" view of what is actually happening ih the class (i.e., 
how. the instruction is being implemented). Such observations dan 
also contribute to the rapport between pilot-test teachers and ' 
project piersonnel. The inclusion of developers and project direc- 
tors in the panel of Classroom observers insures immediate feedback 
from teachers, ai^d two-way communication between the -teachers and 
the project personnel. Of course, classroom observatllons also 
allow for the im%diate identification of problems in the instruc- 
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tional materials. ^ - - ^ , 

^ Teacher judgments are generally seen as an important factor in 
the evaluation of instructional materials^ yet the procedures for 
obtaining these have been relatively ineffective (cf , , Latham, 1973; 
McLaughlin, 1973). The promotion of a high degree of teacher 
involvement appears to be crucial to insuring the effectiveness- oi 
this input source (cf., Baura, 1974). In the implementation of the 
present formative evaluation design with the Mpney, Measurement and 
Time Program, the establishment of teacher involvement insijred. tha^ 
in 97% of the* cases"^^^ evalua^tion forms were completed and returned. 
In addition, the Teacher Review Board meetings- insured that all 
teacher feedback would be obtained, and-tliat any responses to the 
evaluation forms which were not understo^ could, be clarified. Further 
more, the Teacher Review Board .meeting's opened iaiportapt communication 
channels between the teao^^ project directors, and the individuals 
responsible for revising the instruction^ This procedure resulted in 
the compilation of a relatively final summary of the teachers' recommen 
dations for the^ins^ructional product being evaluated. 
' Pr.oject persoiinel can be involved in ^11 phases of the evaiua- 
|ion process. Their opinions, influenced by interactions with the - 

r - • 

either input sources, should be an important input \s^rce to any pro- 
ject engaging in formative evaluation. All project personnel can par- 
ti'cipate in classroom observations, and should continually monitor 
th;e progress of the pilot-test. by investigating ;up-to-date teacher 

evaluation forms, by talking to the pilot-test teachers j and by con- 

j 

sidering the results of data analyses. - -x 

\ ' • m \ . 

2 U Si* 



Allho.ugh most attempts at evaluation Involve data analysis, 
the data collected during formative evaluation should be extensive 
enough to continually reas^jess t;he^prt5gr^siss of Instruction. The 
use of pretestlng^nd^ost^<tlng^on all relevant objectives is^ 
essentia]L..^tirthermor^^ the 'use of more than 4ne measure to eval- 
uate .performance is sdggested (cf., Dershimer, 1968). Frequent test- 
in^^appears^j:^^ effective procedure for continually assess- 

ing the progress of Instruction, as^.well as for identifying forgetting 
effects that the instruction must compensate for and transfer effects 
upon which the instruction can. build. In the implementation of the 
formative evaluation design by the Vocabulary Development Project, 
this procedure Allowed for the cqjnpilation of. test-inferred recommenda- 
.tions for revision, a document which was. extremely valuable during the 
actual revision of materialise*- , ^ t ^ 

The formative evaluation design proposed by the Vocabulary Deveiop- 
ment Proj^t is .an extensive one which is somewhat time-consuming. 
'Yet, the design and the implementation of £fc appear to be extremely 
worthwhile in^terms of the amoi^ and nature of infolnnation obtained. 
'Any project which successfully implements the formative evaluation 
design can expect to obtain the dnformation necessary for a valid and 
•relatively final revision of the instructional product subjected to 
the evaluation. ^ „ ' 

Although the purf)ose of the present paper was to describe the forma 
tlve evaluation design 'ofth^ Vocabulary Development Project, the detail 
of the specific instruments and j^rocedures have not been included here. 
Reports on the formative.; evaluation of ea^h unit (cf . , Krus, Thurlow, 
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Howe, Taylor & Tumure, 1974; Thurlow, Krus, Howe, Taylor, & Tumure, 
1974 a,b,c) provide more detailed information (e|f. , test questions, * 
observation forms, etc.), anci^eports on the summative evaluation of 
each unit (cf., Krus, Thurlow, Taylor, &- Tumure, 1974 a,b,c,d) demon- 
strate how the formative evaluations r.esulted in improvements in field- 
test procedures. , ' • 
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■ Footno'tes ' 



^The formative evaluation design of- the Money, Measurement and 
Time Program iJbok many months to condifptualize and even longer 
to describe for others planning to do formatiA^e 'evaluation. 

'Although gratitude is^ue to all those who participated with " 
the Vocabulary Development Project when in the process of for- 
ma^tive evaluation, special thanks are due to Jenny R. Armstrong 
and Donald Hubbard from the University of Wisconsin at Madison. 
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TIME WITH THE aoCK ^ • » Teacher^ 

Book Two 

School 



Lesson # J. fitle lt.\)i^Ut.. U<^\^l-lm./ » Date_ 



Purpose^ Behavioral Objiectiv'es, Lesson Outline, and Materials >Jeeded 

\ 

Purpose and Behavioral Objectives 

* Were the purpose and objectives state<i clearly enough for you to understand 
the goal of the lesson? 

\ • - None Some Most All 

for how many children m your class were the 
^ purpose and objectives appropriate? { ) ( ) ( ') ) 

How many children did you feel had reached 
the objectives beforfe the lesson was* 

presented? { ) ( ) ( ) ( ) 

Lesson Outline . 

Did" the lesson outline help you in planning. the use of the lesson activities 
and the tape presentation? 

' ■ i 

How could it be Improved to.be morfe helpful to you? 

f 

Mctterials Needed / 

Were; you able to obtain the mat^jials that were required? 



Did you feel any of the required materials should have been provided? 

\ 
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Lesson # 

Pre-Actlvity 

This pre- activity was designed to be structured f.or ycm- 



Was it helpful for the activity 


Yes 


No 


Unsure 






^to be as structured as it was? 


( ) 


( ) 


( ) 


Was the activity structured enough? 


( ) 


( ) 


( ) 


Were the steps in the activity 








clear? Could you see the * 








rational behind the steps? 


( ) 


• 

( ) 


( ) 









Are there any ways in which, you- would change the structure of the 
activity? * 



„ , < , — None Spme Most All 

How many children in your class did 

you feel needed the pre-activity? (• ) ( ) ( ) ( ) 

Of the children who needed the pre- 
Activity, how many benefitted 

from, it? .()()()-() 



What was the effect of the pre-activity oa.ihose children who you 
felt did not need the pre-activity? 



How long did.it take your class to complete the required pre-activityl 



Was the Pre-activity sufficiently explained so that you could direct 
it without difficulty? 



Are there any other activities which you feel should be included as 
pre-actlvities? 
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^ , , Lesson # 

Yes No Unsure 
Did the pre-actl\rity adequately pFepare ' ' 

your class for the tape presentation? • ( ) ( ) ( ) 



Tape Presentation 



Iritroductory , Relation 

^ . .1' Yes No . Unsure 

Did the Introductory relation Interest the' ^ ^ 

• children and get them to look at' the !' ' 
coVer picture? ; ( ) ( ) » X ) 

While looking at the cover picture, did , ' . , f * 

the children attend to what was said?' ( ) C ) , ^ ) 

Did you feel the cover picture was appro^ 

prlate for the Introductory relation • ^\ - 

and the tape presentation as a whole? (* ) C ) O 

Dl(^ the introductory relation succeed in , . ■ ^ 

preparing the children for what the 

lesson was designed to teach them? • C ) C ) ( ) 

Did the introductory relation prepare you 

for the tape' presentation? ' C ) . ( ) ( ) 



Presentation of Words 



Yes No .Unsure 



Were the words presented in the best 

possible order? ( ) ( ) , C ) 

—Did you feel, there was a^smooth flow* ' ^^^^^^ Sometimes Not Usu^ly 
from one word to the next in the 

tape presentation? ' ( ) ( ) j^) 



(Definitions ) : Following the tape presentation, did you feel the 
phildren had obtained definitions for each of the words present^ 
in the tape? 

( ) Yes C ) No 
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Ust each word in the lesson and the type of definition you feel 
most of the^ children in -your class obtained for that word (i.e. 
none, rote, non-generalizable , fuhctidnal, etc.). ** 



(Elaborations); Overall, were the elaborations (stories), distractine 
or helpful to the children?* 

( ) Helpful (■) Distracting ( ) Neither 

List any elaborations which you felt were especially superior 
or inferior. 



(Relations); Did you feel the children understood the relationship 
between the words by the end of the tape presentation? 

■ ( ) Yes ( ) No ( ) Unsure 

Do you think the Summary Relation at the end of the tape presentation 
played a significant role in insuring that . the children understood 
the relationships? 

.( ) Yes ( ) No ( ) Unsure . 

Is there any form (e.g.,, story , -questions , physical activity) that ' 
you think would have made the Summary Relation more effective 
"or interesting? 



List the numbers of the pictures and/or worksheets used during the 
tape presentation and describe their appropriateness (e.g., very 
good, adequate, distracting, inappropriate, unnecessary). 



^ _ How long did it take your class to complete the tape presentation? 
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Lesson If 



) 



Post-Activities 

i 

(General Coitments): Yes No^ Unsure 

Jn general, did you feel that the post- 
activities strengthened the concepts 

developed in the tape presentation? ( ) ( ) ( ) 

Were the post-ac^tiyltles sufficiently 

eaqplalned so th§t you could direct ^ 



t 



them without difficulty? ( ) ( ) ( ) ^ 

Were the post-activities sequenced in 

the best way? , ()()() 
(If not, how would you sequence them?) 



Are there any other activities that you feel should be included in 
the post-activities? ^ 



r 



(Specific Coiranents) : A number of post-activities were suggested to you. 
Please list each activity you used by kind (Required or Optional) 
and number, and give your opinion of the activity and how you think 
it might be strengthened (include, if. possible, the amount of time \~ 
spent on each activity). It is important that we get your specific 
comments on every activity that you have used. Feel free to use 
as much paper as necessary. * 
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Lesson # 



General Comments on the Lessor* 



Please look at the pag^ilTyour Teacher's Manual which shows the 
vocabulary words for ^he unit (page 11), Note the position of 
this lesson within the unit, ' 

^ " . Yes N© Unsure 

• \ Does this chatt help you to, understand 

the place of this lesson /in the whole' 

sequence of the unit? ()()() 

. Do you feel that the children in your 
' - class are now educationally and ' ' <^ - 

motivationally ready- for th% next . * 
lesson in theuQit? / ) ( ^ f ^ 

i ''" Are there any, words that you think the 
j children should have been taught prior 

^ to this lesson? If yes, what are t]iey? (.) ('() ( ) 

] At this point, do you aferee with the - ' 

; ordering of the lessions? ( ) ( j ^ y 

I£not, how would you change the sequence of the. lessons ' <or 
, is there a lesson not included here- that you feel is nee"ded ' 

*j «nd should be inserted before this lesson?). 



Look at the purpose and behavioral objectives for the lesson. Did 
the activities and tape presentation of the lesson meet these =^ 
objectives? 

How many Children did you f^SWew the voqabularj- concepts at the 
end of the complete less on tI-^ 

( ) None ( ) Some^ ( ) Most ( ) All 
Did the children enjoy the lesson? 
^ What aspect of the lesson was most popular? 

What aspect of the Ipsson was least popular? 
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Are there any changes or additions you would recommend to enhajce the 
'children *3 enjoymeSt and/or learning without distra*cting from the 
lesson? 

♦ 



If you had to pick the one aspect of the lesson which you felt was the 
most important In insuring that the, children learned the concepts 
presented, what would you select^i^n this lesson? . 

' I 

V, 



How much actual time did you Qpend on thi^ lessout 
Total niimber of days? 

\ 



Approximate total amount of time? 



What was^your feeling about the length of t^ lesson? -^' 

i ) Too long ( ) t/oo short ( ) About right 
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